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(57) Abstract 



An artificial reflection of an actor or foreground object is created by imaging the actor by a video camera, calculating the position of 
the actor relative to the video camera and transforming the image of the actor when considered as a two dimensional planar image into a 
reflection in a selected plane different to that of the actor or object, said selected plane being a "ground" plane in a combined image. 
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METHOD AND APPARATUS FOR CREATING 
ARTIFICIAL REFLECTION 

The present invention relates to a method and apparatus for creating 
5 artificial reflections and more particularly for creation of artificial 
reflections in virtual studio and blue screen productions. 

Virtual studio technology comprises systems which separate actors 
or objects from their background by keying the actors or objects from the 
10 background according to various parameters including chroma keying, 
distance keying and other keying techniques. Keying the foreground actors 
or objects from the background enables the integration of the foreground 
actors or objects with a different background, thereby enabling the 
production of a much more complex background. 

15 

There are many examples in television production, where live 
foreground video images, keyed by using chroma-key technology are 
combined with previously prepared and stored background images. A 
chroma key studio is characterised by special walls, having a unique mono 
20 colour or a selection of distinct colours. The actors or the objects are 
placed in front of the coloured walls. The overall image is processed by a 
chroma keyer, that separates according to colour difference the actors or 
objects placed in the studio, from the studio walls and produces a 
foreground image. 

25 

The Chroma keyer also combines the foreground image with the 
preferred background image to produce the output image. 
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There are other techniques to separate foreground actors and 
objects from their background, one of them is distance keying. With 
Distance technology, for each pixel in the image a depth measurement is 
performed. By defining a threshold distance it is possible to distinguish 
5 between foreground actors and objects and the rest of the background, 
thus creating a foreground and a background images. 

When performing a simple integration of the foreground image with 
the background image, the integrated image looks unrealistic especially 

10 concerning the lighting and reflection conditions. Due to the fact that the 
foreground image and the virtual background image are taken from two 
separate sources, the lighting conditions are generally unmatched. This is 
particularly true when dealing with the foreground actors and objects 
reflections, that have to take into account the camera position and 

15 orientation related to the preferred actors or objects. 

Reflections presented in a video scene are an integral part of 
realistic imaging. In a normal image the reflection image captured by the 
camera is a result of light originated from foreground objects, then 
20 reflecting on a reflective surface toward the camera. Due to this specific 
trajectory of the light ray, object's reflections are always oriented toward 
the camera, with shape that depends on the object distance from the 
camera, the camera position, the camera viewpoint and the amount of light 
originated from the object. 

25 

The current method for creating reflections in a virtual studio is to 
place a real reflecting surface that enables the introduction of the reflection 
image. This technique is cumbersome and limits the production's 
possibilities. In case of chroma key studio it also reflects colour from the 



WO 00/64144 



PCT/GB00/01282 



studio walls and floor onto the actors and objects, causing a much more 
difficult chroma key separation. 

Another problem related to the current method for reflection is to 
5 cast reflection on additional object positioned in front of the foreground 
actors and objects, or part of the background image placed in front of the 
foreground actors and objects. In that case, the reflected image can not be 
placed in a proper way since it was constructed upon a different surface 
geometry. 

10 

It is an object of the present invention to provide a system and 
method for producing a two-dimensional video image with artificial 
reflection in real time. The invention pertains generally to any video scene 
and particularly to virtual studio systems including real and virtual videos 
15 that comprises background and foreground objects and images. The 
present invention proposes a dedicated technique to produce reflections for 
any object in the scene, given its position in the scene. 

The above mentioned problems, relating to introducing reflections 
20 in a chroma keying environment and other problems are addressed by the 
present invention. The present invention will be understood by reading the 
following specification. A system and method is described for creating 
artificial reflections of foreground objects. 

25 In particular, the present invention describes a system design of a 

chroma keying system, positioning system for any foreground object, 
camera parameters (x,y,z position, pan, roll, tilt, zoom, aspect ratio) 
capturing system for every studio camera. The generated reflections will 
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follow both object movements in the scene and the camera movements, 
resulting in an high quality look of the integrated image. 



A system is also described where the reflection can be reflected 
5 from the studio floor, walls or any computer generated object located in 
the scene that has flat surfaces. 

In addition the present invention can be applied simultaniously to a 
multi-camera studio. 

10 

Thus, by capturing the position of the object/actor, the camera 
parameters (especially the camera position and point of view), the 
object/actor foreground image, it is possible to calculate a reflection 
image. Because reflection is caused by light rays that are starting from the 

15 object then reflecting from the surface to the camera one has to know all 
the parameters regarding the position of the object and the camera 
parameters, it is not possible to construct the reflection image if one of 
this parameters is missing. This is what makes the present system special 
since the systems captures both the object position and the camera 

20 parameters. 

The present invention provides an apparatus for creating an 
artificial reflection of an actor, said apparatus comprising a main camera, 
an actor position tracking device a system for tracking camera parameters 
25 (x, y, z position, pan, roll, tilt, zoom, aspect ratio) and a transformation 
processor, said transformation processor comprising means for imaging 
said object from the position of the main camera, means for approximating 
the image of the actor to be a two dimensional image facing the camera 
and means for transforming the two dimensional image to produce a two 

4 
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dimensional reflection on a surface adjacent to the actor, means for 
generating a substitute background image and means for combining the 
image of said actor, the two dimensional transformed reflection and the 
substitute background image into a combined video output in which both 
5 said image of said actor and the reflection of the actor are present in the 
combined video output thereby providing a realistic association of said 
actor with said substitute background image. 

In the present invention the term actor is defined as including other 
10 foreground objects. 

The invention also provides apparatus for creating an artificial 
reflection of an actor in which the transformation is a perspective 
transformation. 

15 

Alternatively the transformation is a geometrical transformation. 

The present invention also provides apparatus for creating an 
artificial reflection of an actor as in which the actor is captured and 
20 separated from a background by chroma keying, edge detection, 
difference keying, luminance keying image triangulation and/or depth 
keying. 

The present invention also provides a method of producing an 
25 artificial reflection of an actor, said method comprising the steps of: 

(a) obtaining an image of the actor by means of a video camera; 

(b) tracking the position of the actor relative to the camera; 

(c) tracking the camera parameters; 

(d) transforming the position of the actor by the step of : 

5 
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i. processing the image of the actor formed by the video 
camera as a two dimensional image; 

iL transforming the two dimensional image of the actor 
to represent a reflection of the actor in a different plane to the plane 
5 of the two dimensional image of the actor, said different plane 

representing a ground plane as defined on which the reflection is to 
be present; and 

(e) combining the video image of the actor, the transformed 
reflective image and a virtual background image to provide a combined 
10 video output comprising a real actor, a virtual reflection of said actor and 
a virtual reality background. 

The invention also provides a method of producing an artificial 
reflection of an actor in which the transformation of the two dimensional 
15 image of the actor is a perspective transformation. 

Alternatively, the transformation of the two dimensional image of 
the actor is a geometrical transformation. 

20 The invention also provides a method of producing an artificial 

reflection of an actor in which the image of the actor is captured and 
separated from a background by choma keying, edge detection, difference 
keying luminance keying, image triangulation and/or depth keying. 

25 The term ground plane is defined as any plane adjacent to the actor 

which could include walls or computer generated objects. 
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In order to add an artificial reflection to a video image, the 
foreground video must be captured. It will then be composed into the 
image in a method that is described below. 

5 The present invention will now be described, by way of example 

with reference to the accompanying drawings in which : 

Figure 1 shows diagrammatically apparatus illustrating production 
of a flat reflection of a foreground object; 
10 Figure 2 shows a first embodiment illustrating in greater detail a 

first apparatus and method for production of a reflection of a foreground 
object as illustrated in Figure 1; and 

Figure 3 shows a second embodiment illustrating in greater detail a 
second apparatus and method for production of a reflection of a 
15 foreground object as illustrated in Figure 1 . 

In order to capture the chosen foreground object(s), the object must 
first be captured by a video camera. In order to capture and separate the 
object from its background, the system must use a keying method. The 
20 keying method is preferably chroma keying normally used in virtual 
studio, but can also be done using edge detection, difference keying, 
luminance keying, image triangulation and depth keying. 

In order to introduce an artificial reflection of an object in a virtual 
25 studio environment, the video image should be segmented into three parts; 
the object as it appears in the original video image, the reflection as a 
mixed foreground and background image and the rest of the video image 
as background image. The level of mixing the reflection image with the 
background can varied by a separated control unit that can be adjusted 
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according to original light intensity or the distance which the light ray 
travels from the object to the capturing camera, or any method suitable to 
form the proper lighting intensity of the reflection image. The level of 
mixing can also varied inside the reflection image according to lighting 
5 condition, position of the original point in the actor or/and object two 
dimensional image and intensity map design by the operator simulating 
surfaces with varied reflection coefficients. The background image can be 
computer generated in cases of a virtual studio scene and can be composed 
of a number of sources, such as graphics or video images. 

10 

In order to calculate the proper shape and texture of the artificial 
reflection cast by the object as should be seen from the point of view of 
the video camera. The method proposed in this application for calculating 
the artificial reflection is based on the manipulation of the original 
15 foreground object image. In this application we approximate the 
foreground image of the actor or/and object to be a two dimensional image 
facing the shooting camera. 

The reflection image calculation is constructed from several steps: 
20 First, detecting the actor or/and object position, this can be done by 

several position detection methods like the one presented in International 
Patent Application No. WO98/34410 "Virtual Studio Position Sensing 
System". 

25 Second, detecting the camera parameters (x,y,z position, pan, roll, 

tilt, zoom, aspect ratio), this can be done by any camera parameters 
detection system, like the one presented in International Patent Application 
No. WO95/30312 "Improved Chroma Keying System". 
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Third, keying of the foreground object from the rest of the video 
image, using any keying technique, preferably chroma keying which is the 
conventional keying method in virtual studios. 

5 Fourth, producing the reflection image by calculating a geometrical 

transformation of the foreground image according to the object position 
and the camera parameters. The calculation of the reflection image by 
geometrical transformation is aplicable due to the assumption that the 
object is presented by a two dimentional plane facing the shooting 

10 camera.. With out this assumption there is a need to calculate for each 
point in the foreground object the exact position in space and perform a 
separate transformation calculation for each point. 

Fifth, combining the foreground actor or/object with the reflection 
15 image to the background image. 

With reference now to Figure 1, the apparatus 10 comprises a 
foreground actor 12, a chroma key colour background 14 and floor section 
16. A main video camera 18 is situated to film the actor 12 against the 
20 chroma key coloured background which may, for example, be a blue 
colour. 

In Figure 1 a reflection 20 of the actor 12 is shown which in the 
normal image will be present in front of the actor showing the reflection 
25 - of the actor, or foreground object of the floor 16. 

In a virtual studio comprising, for example, a blue screen studio the 
reflection 20 does not appear, since the floor is not reflected or has very 
poor reflection properties. Thus, an artificial reflection should be 

9 
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constructed. Figure 1 shows how the reflection should be in a real image. 
Directed toward the camera with shape dependant on the object/actor 
position and the camera parameters. Figure 1 does not resemble a true 
image of combined object and reflection image, since it is a side view. 

5 

With reference to Figure 2, a first apparatus is shown which relates 
to a more general technique which keys the foreground object out from the 
background image and then performs a transformation to calculate the 
reflection image. Then the object/actor + the reflection + the artificial 
10 background are summed to form to output image. The apparatus 
comprises the main camera 18, an actor/object position tracking device 
200 such as described in co-pending International Patent Application No. 
98/34410 (Virtual Camera + LED), a chroma keyer 202 such as 
manufactured by ULTIMATTE (Registered Trade Mark). 

15 

The device 200 and keyer 202 are both connected to a 
transformation processor 204 which also has inputs from a camera 
parameter tracking device 206. 

20 The output of transformation processor 204 is connected to a 

combiner 208 in which the output of keyer 202 is combined with the 
output of the processor to provide a combined image which includes the 
reflection 20 provided by the transformation process. 

25 This combined output from combiner 208 then requires to be 

further combined in a combiner 210 with a background image provided by 
a camera or computer 212 which provides the "virtual" background image 
which when combined with the output of combiner 208 provides the 
composite video output via output circuit 2143. 

10 
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The key element to the invnetion is that by knowledge of the 
position of the actor/object 200 and the camera tracking device 206 and by 
consideration that the object comprises a flat surface the reflector can be 
5 accurately processed in relatively simple combiner circuits. 

With reference to Figure 3, an alternative circuit is shown. In this 
second example (which is possibly the more practical method), the image 
from the main camera is combined with the transformation of the same 

10 image (corresponding to the reflection transformation). The integration is 
done in a specific way. The reflection image is integrated into the original 
image in areas where it is shown that the foreground object does not 
appear. This can be done, since we have general knowledge of the object 
dimension and we know its position and the camera parameters. After 

15 making this integration a chroma keyer is used (by that saving the object 
keyer used in the above first example) which also adds the background 
image. This circuit is similar to that of Figure 2 and units within Figure 3 
which perform the same or similar function to those in Figure 2 are given 
the same reference numerals. 

20 

In Figure 3 the transformation processor 204 operates directly on 
the output of the main camera 18 without the signal having been separated 
into foreground and background. 

25 In Figure 3 the combiner circuit 210 of Figure 2 is replaced by a 

more complex combiner unit which combines the function of both a 
combiner and a chroma key circuit. Thus, the function of units 202 and 
210 in Figure 2 are combined into a single unit 216 in Figure 3. 
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The output of chroma key combiner 216 represents the video output 

214. 

In the Second specific embodiment presented by Figure 3, the 
5 system is constructed by a Chroma key studio, capturing video camera, 
position detection system to capture the position of the actor or/and object, 
Camera parameters detecting system, processor to compute the 
geometrical transformation, computer of video source for the generation 
of the background image and a chroma keyer. The generation of 
10 reflection image for the present embodiment is constructed by the 
following steps: 

First, an image of the actor or/and objects is captured in a chroma 
key studio. 

15 

Second, a selected part of the image that includes the actor or 
object is then transformed by geometrical transformation performed in the 
processor using actor or object position and the camera parameters. 



20 Third, the transformed image is then mixed with the original image 

in a proper way. 

Fourth, the mixed image is then combined with the background 
image using a chroma key combiner. 

25 

In a specific embodiment the geometrical transformation to form 
the reflection image is a perspective transformation. 

In another embodiment the geometrical transformation to form the 
reflection image can be any geometrical transformation. 

12 
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CLAIMS 

1. Apparatus for creating an artificial reflection of an actor, said 
apparatus comprising a main camera, an actor position tracking device, a 
system for tracking camera parameters (x, y, z position, pan, roll, tilt, 

5 zoom, aspect ratio), and a transformation processor, said transformation 
processor comprising means for imaging said object from the position of 
the main camera, means for approximating the image of the actor to be a 
two dimensional image facing the camera and means for transforming the 
two dimensional image to produce a two dimensional reflection on a 

10 surface adjacent to the actor, means for generating a substitute background 
image and means for combining the image of said actor, the two 
dimensional transformed reflection and the substitute background image 
into a combined video output in which both said image of said actor and 
the reflection of the actor are present in the combined video output thereby 

15 providing a realistic associate of said actor with said substitute background 
image. 

2. Apparatus for creating an artificial reflection of an actor, as 
claimed in Claim 1, in which the transformation is a perspective 

20 transformation. 

3. Apparatus for creating an artificial reflection of an actor, as 
claimed in claim 1, in which the transformation is a geometrical 
transformation. 

25 

4. Apparatus for creating an artificial reflection of an actor, as 
claimed in any one of claims 1 to 3, in which the actor is captured and 
separated from a background by chroma keying, edge detection, 
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difference keying, luminance keying image triangulation and/or depth 
keying. 



5. A method of producing an artificial reflection of an actor, said 
5 method comprising the steps of: 

(a) obtaining an image of the actor by means of a video camera; 

(b) tracking the position of the actor relative to the camera; 

(c) tracking the camera parameters; 

(d) transforming the position of the actor by the step of : 

io i. processing the image of the actor formed by the video 

camera as a two dimensional image; 

ii. transforming the two dimensional image of the actor 

to represent a reflection of the actor in a different plane to the plane 

of the two dimensional image of the actor, said different plane 
15 representing a ground plane as defined on which the reflection is to 

be present; and 

(e) combining the video image of the actor, the transformed 
reflective image and a virtual background image to provide a combined 
video output comprising a real actor, a virtual reflection of said actor and 

20 a virtual reality background. 



6. A method of producing an artificial reflection of an actor as 
claimed in claim 5 in which the transformation of the two dimensional 
image of the actor is a perspective transformation. 

7. A method of producing an artificial reflection of an actor, as 
claimed in claim 5, in which the transformation of the two dimensional 
image of the actor is a geometrical transformation. 
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8. A method of producing an artificial reflection of an actor, as 
claimed in any one of claims 5 to 7, in which the image of the actor is 
captured and separated from a background by chroma keying, edge 
detection, difference keying, luminance keying, image triangulation and/or 
5 depth keying. 



15 
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